Yeah. So this was that face to face meeting. Um but it uh it doesn't There isn't th so there are things in the eye tracker record that we definitely don't want in the G_D_F_ format, um like the frame rate eye movement. You know, we we uh uh i we're if we're expecting the G_D_F_ format to be um sort of a parsed version of events. And you know there's just too much raw frame f frame rate stuff coming out of this. You wouldn't prob you wouldn't want it in this kind of format for up-translation to the other things. Yeah, we yeah, you can get back to it. Yeah. We're not throwing anything away. This is just the stuff Yeah. Yeah. But the th but the way to think of this is this is the data that we want to be able to analyse against the other tracks of data. So the data that we wanna compare with a language or with whatever. And um so Yeah. S so in essence what you would then do is Well we're not doing frame rate at any frame for well, maybe we are. I don't know. I I had thought we were yeah, the eye tracker does. But But right. W what w um the kinds of events that we had r uh before talked about putting into the record for then use in ELAN and N_X_T_ isn't based on saying every X_ seconds something. You know, give me give me what's happening every X_ seconds. It's more like, you know, give me the fixations and the um the blinks. Yeah. Oh yeah. Durations are in there. But it's not there's a difference between um saying that something is in a particular state every frame, whatever the frame rate is, you know, ten seconds a minute. You know, that's that's one kind of w way of looking at data, and um a parsed version of the data, which isn't at um any particular length. It relies on this really close frame rate that's underneath. No, no. The time the time is still in there. But you'll get uh what you want is to figure out what are the um the concepts behind that data that you want represented. So rather than saying the eye was at this particular place, you know, here, and then a tenth of a second later here and then a tenth of a second later here. You say there was a fixation from this time to that time and there was blink from this time to that time. And Okay. Mm-hmm. Right. So there's two issues here. We're not throwing anything away. But the question is what tool would you use to get that information out of the data? Would you be so the more you can plan Mm-hmm. Yeah. Mm-hmm. So you need to know the percentage of time it was on during that fixation. Yeah? Okay, so the percentage is i i is a cut it's not something you're using for measurement. It's a cut-off for whether or not it counts as a fixation? Is that Mm-hmm. Mm-hmm. Mm-hmm. Right. Yeah. 'Cause all you wanna know is whether they were looking at the same object in the period after. Yeah. Okay. Can we draw it to make sure that that um that this is satisfying Ellen's concerns? Well, there's a white-board, right. We love this thing. See if the pens work. So um this is um A_'s eyes, right? And B_'s eyes. And I think what you're saying you should probably draw this right, is that they you know, if um from this time to this time they're looking at triangle one. W what you're saying is that you want to know in the critical you wanna know after they're they're looking at triangle one what's happening in this period with this guy, right? Mm-hmm. Mm-hmm. Mm-hmm. Mm-hmm. Mm-hmm. Mm-hmm. Yeah. So Mm-hmm. Alright. So Yeah. So, you know, he might get triangle one there and no maybe this one's gonna overlook lap quite a bit. So what you want to do is be able to define um bigger periods, which is the period when they're sort of interested in triangle one overall, right? This is sort of some meta level analysis. Yeah. But that's that's not a something that you can do in the N_X_T_ query language, but it you can't do that in any you know, this is such special purpose s but it's easy enough given the data format for any of these things to do that. So you could do it on 'cause i it's just a matter of i the hard part is deciding how close together these have to be before you decide that this is a event that you wanna pull out. Because, you know, algorithmically you're you're already putting together I mean this is essentially already, you know, these fixations with stuff in between. And uh um, you know, we we've got an algorithm for deciding when that's a look, I guess. You already have that or no? Oh yeah, yeah. So it's all fixations. They're these are all fixations and saccades, but within the yep. So that's easy enough. So the hard thing is they could be moving their eye arbitrarily here, right? And you know, maybe even having fixations on other things? Yeah? Yep. Back and forward. Yep. So um this isn't yeah. But what you need to do is we we can already build into this G_D_F_ format these regions if you can give a definition of what you think that is. But I think the right way to do this is to be able to inspect the data in um some tool and play around with the definitions. Because you won't get it right the first time. Yeah. Mm-hmm. And uh ELAN's a reasonable choice for that, right? What you need is something well no, because you need to Y you need to be able to see the effects of this. So I g I guess this is a case of Craig writing some scripts so Robin having some ideas about what the relationship is and saying, you know, add this data a automatically. It I mean it's not that hard, right. It's just there'll be a bunch of these, you'll have some rules about how maybe how long you spend on other objects and w whether the other object is um the existing construction or not, right. And so um Ah, right. So if they if they uh they had a fixation on C_ you wouldn't be worried. But if they had a fixation on Yeah, they're yep. But yeah, they're gonna look at the clock um and they're gonna look at they may look at completely irrelevant Mm-hmm. Yeah. But that's like a separate analysis, right. So It no no. The so every separate Yeah. Mm-hmm. Yeah. But what we find e what we can get out of the data easily at the moment is at this kind of level, right. And then it's a case of defining algorithmically all these other transformations that you want. So this is one uh we hadn't thought of adding before, but we should, right. Which is um well, how would you decide whether something was the current addendum uh the cu So that's a human decision. Yeah. Um But the ex But we believe But we believe J_P_ to be defining that maybe. Yeah. Mm-hmm. I could believe them, yeah, no, taking an interest in that. I mean the general division was language here and other stuff there, right. I'll just I'll just minute that. Oh yeah, eye tracker here. Yeah, I mean we know that. Well yeah. But I mean with transaction coding on something like this you would use i y you wouldn't consider it verbal analysis exactly. It'd be verbal plus action. Because it's a task breakdown. It's just a segmentation. Yeah, but you wouldn't have you you wouldn't do two codings for task level one you wouldn't do transaction coding in a action segmentation. I I don't think that's realistic, because I think what you'd actually do is use um the full record of what you have and do a segmentation. I think it'd be hard to understand what was going on if you used like just the language without watching the video to decide whether we're breaking down the task. Oh, you don't want it to be coding th uh based on that though and then that seems a strange way to go about things. I mean this the language events are multi-modal, right. You know, they're they're doing all these things together. Wha why do you care whether they can do it just based on the the language? Is Well, it seems to me much less important than the other things one could get out of this data. But I, you know, I it's a possibility. We'll Yeah, well I c I can see maybe wanting to know whether you can do the chunking just based on the actions without the language in cases where they use both. It's more g I can't quite see why it's important to know that they can do it just based on the language when w you know that they had both. Uh-huh. Well, I think that one's just bound to fail. But yeah. Really? Okay. Anyway. So back back to this main problem, which is yeah, w the record that we are getting in the first instance is about the m about these you know, they're looking at some region of the screen defined dynamically. And then we need some way of knowing what the you can you can see adding these other analyses about, you know, they're they're jointly focused on this region and trying to figure out what s percentage of time they're looking at it as adding new tiers of information to either the ELAN track or N_X_T_. They're they're both sort of track based in this way. Um but the hard part is knowing how you wanna do that. And I I think in the first instance what we're trying to get into the G_D_F_ format is just this. And then we've got no option but to uh figure out a way for like to look at this, just explore this data, suggest ways about doing it and um be able to play them back and see um when we think we've got these things right. I don't think we even know what the set of these things are that we want, much less how to get 'em yet. So th Yeah, that's right. So this is not the you know, this is something we're aiming for. But this isn't something that affects what Craig programs for the initial G_D_F_ translation. Um Yeah. So let's just return to this question of frame rate. Because this is the thing we were not planning to um transfer into this format. So again, you know, what a frame rate kind kind of coding is ag it can again be seen as a track. But it says it's a coding like this, right, where you say it's in state A_, state B_, state C_, state A_. Right. So you right. So I'm just you're happy that it's not gonna have frame rate uh like this. It's gonna have an interpretation like this. What does that mean? Sorry. Right. W right. But it seems to me it seems to me that Ellen's concerns might mean we need to add more information to these tags because um, you know, this thing in itself is th bunch of fixations, right, with saccades. Fixations and the percentage of time those fixations cover, you might want. Okay. So number of fixations. Percentage of No no. This is this is still um uh summary data. They're he's we're talking about just adding attributes to these things that say the number of fixations that counted as that looking at triangle one, right. So y it's possible Yeah, the spots'll be different anyway. 'Cause i you're not gonna g have the same pixel. So um we have the option that we can put in an A_ fixation thing here, right, as well. If you want the smaller if you want the smaller coding in in here. So that it's not just parsed into this idea of which object. But you also have the raw fixation um data. We can put that in as another track. If you think that that's something that you might want to look at in one of these tools that shows you the tracks against each other. Yeah, I d that's all he asked for. Yeah. Well, I think Yeah, and in N_X_T_ there's no cost, right, because the y the uh you just don't choose to load those. All we're doing is dumping it as output that you can load if you choose to. Um i g s you know, so if they're things that you just know are crazy and you're not gonna want then we don't do it. But otherwise we go ahead and dump 'em. 'Cause it's easy to dump out the fixations. Um yeah? Okay. So it's the number of fixations the did you want the you M not e well, if it's each, you have to d you have to break it down into another tier, right. You can't you can say there were three fixations and they averaged uh a certain time. But you don't wanna D well, what about the the overall sum of the durations? Is that I mean which way do you want to Yeah, then that's that's the way you want it? That way around? Okay. And uh y did you want the percentage of time fixated as opposed to which is sort of another view of this number. You can have as many of these as Yeah. Well, it's not um the the extra cost of having these as attributes is not high. So um, you know, it saves you i uh having to do write scripts to do arithmetic on that later if you know that these are numbers that are gonna be useful to you. Mm-hmm. Yeah. I don't know. Well well but y those are different tracks of information. So what you want is one track of information about which object it's one, one about whether or not it's on the other guy's gaze, and one about whether it's on um one of the mouse positions. Yeah, but i in in the analysis you should treat those as completely separate tracks of information. Because they can be looking at they could look sim simultaneously at somebody's mouse, somebody's gaze and some object, right. So Yeah. So the these are independent. So so what we want is whether they're looking at s at the other person's gaze, right, and say they are in this period well, I mean uh they're probably not given what I've trying to Mm-hmm. Mm-hmm. Mm-hmm. Yeah. So g so there's there's a lot of tracks here, right. Because well we just added this fixation track. And then there is A_'s mouse, not their eyes, is on an object, right. And the mouse might be on triangle one. But it doesn't matter. I mean i the mouse we're just saying where is the mouse. You know, it's on this object. And then um A_'s eye is on well A_'s mouse I guess, right. A_'s eye might be looking at A_'s mouse here. A_'s eye might be looking at B_'s mouse here, right. I mean these are all the different ways of taking cross-products at the things that could be co-located. So you just treat 'em as independent. And I'm guessing from what you say yeah. I guessing from what you're saying you want all these tracks. Yeah, which is fine. There's no problem with that at all. Yeah. S so um uh you kinda got the idea here? Yeah. But the important thing is you don't treat that entire set as mutually exclusive and exhaustive. It's just so tha the objects is like a separate level of analysis. If whether they're looking at the objects than whether they're looking at the gaze. 'Cause they can happen at the same time. Does that happen? Oh, right. Uh-huh. Okay. So what you're saying is like a new problem for us, which is I'm gonna change That's the bad one. Yeah. So you're saying Okay. Can you write on my pen pad? Just say, yeah, pens. White-board pens. So you're saying, you know, they could be looking at um uh square one at this point. 'Cause they close together. Okay, that's a problem for the data models and either of the things that we were using. What do you mean by greatest overlap? Draw uh my g my uh spacial reasoning is no good. See uh I I'm the one that uses the white-board. 'Cause I actually put the the microphones on. Uh this is the better pen. Uh-huh. Yeah. But you had a way of choosing which one, right? So you just mean which i is it closer to the is the circle I mean, what is this circle? They're looking at a pixel or something, right? Right. S right. Oh, so those vary in size. Yeah yeah. Mm-hmm. Yeah. They could be. Oh, but what pain that's gonna be analytically. Because it's gonna how many parts are there? Like Okay. I know ways of getting around this analytically. 'Cause you c If they're really close to each other, it's gonna get foxed. Yeah. It's not the data format I'm worried about. It's the um the way you do the analysis. Because you don't wanna have to say um, you know, did they jointly look at triangle one. Okay. Did they jointly look at square one. Okay. Did they jointly look at you know, you need some way of uh uh going over the whole thing. But i c we can find ways around that. I think for now we we do it this way and then we we think about what we need out of it in the end. It's just you know, in ELAN it that has the side effect that if you go for the naturalistic way of of up-translating to ELAN, there'll be a zillion tracks. And it'll probably ruin their viewer. 'Cause you'll get very sparse data on each track. It'll be like the old referring expression generalis uh uh visualization um the map task. You know, where they talk about one object one landmark and then another landmark. Well, it's big, right. Well we'll see if ELAN likes it or not. Yeah, and we b we better also like have in the representational list of par like even in the N_X_T_ up-translation a list of parts. It it affects the way we do the N_X_T_ u up-translation and make things easier for people. Well the And it suddenly becomes and there could be more than one construct at a time, right. It's there is no the construct, right. What why does it why does construct one become construct two when you add something to it? Yeah. Oh, and this is because we don't want it to stay triangle one when you start with a construct it's not a construct at all, it's just a one thing. And we don't want triangle one to suddenly have bigger stuff. Yeah yeah, there are. There are. Mm-hmm. Yeah. That was the next question I had. Yeah. Oh yeah. Mm-hmm. That'll cover most of 'em. Mm-hmm. I yeah. It's triangle two. You don't because that's you'd have to have human coding to do that. Yeah, there might be multiple triangles that you've broke and you won't know which one they're planning on using it for. Well you know which mold they come from. And so you know which shape they match. But you can't possibly know which one of the ones they're meant to replace. Oh, so there's no two parts with the same shape? Then Oh same shape and colour. Right okay. Oh okay. So you can tell what it's meant to replace then. Because it's the same it's uh just a case of the mold, right? yep. Yeah. But the this also Mm-hmm. Mm-hmm. Yeah. We in when you specify these things, do you specify the molds or does it figure out what the molds are based on? So is this hard-wired into Tim's program? Or is it just an artifact of the way we s Right. Okay. So it's hard-wired. So minor modification needed for Mm-hmm. Uh but what were the req wha Has multiples, yeah. Mm-hmm. Mm. Yeah, uh but they don't have enough to make unique Mm-hmm. So what Yeah. Mm-hmm. So I th I think we've gotten to the point where, you know, to try to summarise, uh we we think Tim's program can do both of these conditions that you want. But I'm still worried about what the uh effects are for analysis. Because you were aiming at something in um the G_D_F_ format and I wasn't quite sure what. And w Yeah. So so okay. So I'll summarise what we said about that part that um every time you cast off a new part from a mold or, you know, every black triangle has a different I_D_. And you know it's a black triangle 'cause you know which mold it came from. But you don't know which part it was intended to replace on the screen, because you can't mind-read. What do you mean define? Mm-hmm. Mm. I think that this is a human coding that is part of the action coding. Because I don't see how anybody but a person watching this can guess why they cast off this thing at this time. Yeah, but that's okay 'cause they all have different underlying part numbers. And when you say what a construct No. No, no. Yes. Mm-hmm. Yeah. So we can easily look at things like that automatically. It's just the why they g why they get new parts that's the problem for us. We can't if they suddenly if they decide to get a part of a mold we don't know we can't know why until they do something with it, if they do. Oh yeah, you said that. Um Yeah, but you can But you can break you you can but you can break one when you break one, you don't cast off the mold yourself, it happens automatically? Oh okay, never mind then. So they just have to use it, yeah. Uh-huh. Oh, but they have to drag it up past the mold line. Yeah. Mm-hmm. Yeah, where there's extra parts. Yeah. Okay. So this why question doesn't even arise because 'Cause you know what they're gonna do with it. When they decide to move it past the mold line, they're p doing something with it probably and you'll know what they're gonna do with it because They do it. Yeah. How would you do that? Yeah, but there's there'd be no way of coding that automatically, right. How could you possibly do that? But that's a human coding, right. Mm-hmm. So w So uh that's gonna particularly be a problem for J_P_, right. Remember m all the things we're looking at are f with language with and without language. And it's gonna be very difficult to tell whether they broke something intentionally without the language. Which means it's probably not Yeah, okay. But th um this is properly part of the action coding. So we should well we should make a wish list, right, as a side effect. Mm. they could discuss deciding to break something and and then one guy does it. Yeah, yeah. Mm-hmm. No, no. It was them just dr uh forgetting to um define an attribute they had to Well, whi which is probably a documentation fault, right. Yeah. Yeah. Is there a y yeah. Well, while we're on past things, is there anything else you wanna tell us about the prior part of this, which is the, you know, what else is he holding you up from Uh-huh. Right. So it doesn't slow down the eye tracker. It doesn't do anything nasty. So we we p so we pay for it and Nothing important and noth nothing big enough they're gonna look at it, right? Yep. Yeah. Yeah, yeah. I know. Oh, do you need two copies of Camtasia to do that? So you're gonna look into this, right? Well y c look at the licence conditions, it might not I c I never pre-judge licences until I've read 'em. So uh well and um hold on, before we get go any further. So this has implications for data storage, right? Because w um it means we've got well no, this is three times, right, 'cause we've got oh no, we're gonna build we're gonna use JavaScript to build the videos on the fly, right, 'cause it's fast. So the the permanent storage is just the Camtasia stuff. So uh is that a problem? Where are we putting things? Oh we can we can put 'em in the same thing, right? You can dump 'em into one video frame if you don't mind losing resolution. But that's maybe a problem, right, because So the synchronization's gonna be a problem. Yeah, before the task. No. So These are ba these are back-ups. Right. So the flash get allows you to hand-synchronize them later if you need to by stripping extra video off the front. As long as you make sure you start 'em before the flash then you're fine. But it will there's a a cost of having to go to back-up, which is synchronizing them. Oh okay. So you so you should start the Camtasia really early then. Right. Okay. Uh-huh. That's not where you're putting it. I mean u Well okay, you've got the proper sound record. But you're not dumping it through Camtasia to record it, right. You're What before we were gonna use Camtasia at all, what well what we were gonna do what were we going to do with the sound? Uh-huh. So uh ha wou is there any sound degradation that comes about from putting this through Camtasia rather than running it out straight? Okay. So what you're planning on doing it is bunging it on one of the Camtasia tracks and then splicing it off the Camtasia track. It's an easy bit of yeah. And then you're gonna bung it on the other Camtasia track too if you need it, right? But we'll just we're gonna store all three separately then. And the sound is going to start at the same time as one of that arbitrary one of the Camtasia videos. But you'll know when the real experiment you'll know the relationship between that and the eye tracker timings? Yeah. No human intervention required. It's time-stamped. Yeah. But ho hold on. Do we need that? I mean it I I thought the way that Tim had this set up originally, um the audio uh w the time stamps used there would be joint time stamps between the audio and the eye tracker, so that, you know, uh the audio record ten seconds in was the same as ten seconds into the eye tracker record so that we didn't have to do any extra hand work, you know, any uh chopping the starts of audio signals to get there was nothing okay. So we're relying on your MATLAB script to get us the chopped version of the audio, right? Right, so we can either adjust the eye trai track data or we can actually chop the audio signal to take off the first whatever, however many seconds. Yeah, but uh but d um the uh lining up the Camtasia two is gonna take we w he's working on a MATLAB script for the sound but not for so the one that's got the sound on it, that'll also tell you where to chop the video to get it to line up. The one that doesn't have the sound on it, it won't. Oh. So it's just got the bleeps. Oh okay, so w no hand synchronization required at all. Um as long as the scripts work, right. 'Kay. How confident are you about the finding these things? So we just have to w hope we don't have an any subjects with the odd vocalizations. Yeah. Okay. Yeah. 'Cause this this is the drop-out test. Yep. Okay. Yeah, yeah. Mm. Oh yeah. I know but so you g you got a solution for that then? Oh right, okay. Yeah. You're sure there's not a better solution that involves 'Cause, you know, people might wanna use this for speech recognition or something. You never know if it's the data is there then You know. Yeah. And you're gonna want forced alignment. Oh no. Well no no. Better to collect and sort. Uh well it's the mono Wait wait, w I I thought the sol that a better solution was fixing this problem with the mono microphone socket. Sound card? So you said the problem was the m it the m m i is expecting a mono microphone input. So is that fixable? Like is there any way yeah. Uh I think a new sound card sounds a better solution than monkeying about with trying to filter white noise out. Which isn't hard. But hi but it's only got one input, right. I believe it's easy to put two monos into a stereo signal. Yeah. So Yeah, yeah. So is there any reason why we wou don't do that? Yeah. Well, it's in the old days we probably would have want a little if they'd been in the same room we would have had a little stereo thing in the background as backup as well just to make sure. 'Cause it's slightly dicier you know, if his scripts don't work then you don't get the overlap. It doesn't have to be a real trial. It just has to be two guys talking in the two bits. And then We've both been so badly bitten in the past. Yeah. But this this sounds reasonable to me. I so I think Yeah. Do do they both have the same sound card problem? I wouldn't count on it. Okay. Oh Well uh yeah. I think tell the guys. To yeah, yeah. Yeah. Mm-hmm. Yeah, it's just the way they wired it. So um it's getting late and I've got a two o'clock and I've got a nice lunch. So I wanna eat my lunch today. Uh what did you wanna get through today more? I mean I think I think it was useful going through your expectations about this. 'Cause um that's quite a bit clearer in my head at least and but I think mostly we leave you to go away and re-design the G_D_F_ or add new bits to the G_D_F_ along these lines. And then w Yeah, yeah, yeah. But I think the thing to do is um for you to go away and think about it this way with the different tracks for the different objects and the um the kinds of things we've added about whether they're looking at the other person's gaze and what have you, and and try to change the spec so it it reflects this and maybe um oh maybe by that point there will be some sample data or something uh and the right thing to do is for us to look at the sp the spec the way you understand it now and some data. And then Ellen'll have new ideas about what's needed or about these post-analyses, you know, the and then and we can add 'em in. But it's gonna emerge over time. I mean that's clear. Oh, so the spec to the other conversation, yeah. Yeah, that's that's solved now. Yeah, test. We have to test. Well you don't have your models, or do you? Yeah. Yeah. Well, uh anything you actually need fixed. You know. Joe doesn't live here anymore, right? So Well, i but he wasn't gonna be booking any time to us after the thirty first of October. So um that that means that any bugs that need fixed are now Craig's responsibility, right? You don't know have any change on that, right? 'Cause he that's what he said to us originally. So I would assume, you know, no more contact with Joe. It's not fair to get work out of somebody for free. So you know, ask Craig instead. Yeah. I don't think Well w he wasn't paid in advance. He's being paid in arrears. So it doesn't matter. Uh I think it's I think it's better to have all the software. I mean it's not big things, it's just maintenance at this program at this point and Yeah. I if he can give it, you know, like because the problem is you gotta have an immediately when you discover a bug in that. And we can make Craig shift all his other priorities to do this 'cause we bought him. Right. Yeah. So it's a kinder way. It yeah. Yeah. But it sounds like you don't think there's anything that actually needs fixed at the moment. Because w we only need this to work well enough that you can use it for something and you got work rats. So And how many models do you need to completion? And how long does that take? And you got And you know about complexity? You're waiting on Marloes. You can cold call her today, this afternoon. Mm-hmm. That's her job. Come on. Oh yeah. Okay. So you can dash in there for half an hour at a time? Doesn't sound very useful for you. Uh-huh. Okay, well. You guys sort it out. Mm-hmm. Right, but it would be kind to tell her ahead of time which of your morning sessions you're bumped. Yeah. Mm-hmm. I thought that was the problem that your mornings weren't sliced off on the booking system. 'Cause you know, i if you're sa told use the booking system Yeah. And what are the instr what are the people upstairs downstairs have um i you know, as rules about who you hand the keys to. 'Cause, you know, we like people to know to ground the oh I thought the the lending for the booking system Caroline. See, 'cause you're supposed to uh have training in Oh okay, so that's the safety. And you control those accounts. Oh okay, that's fair enough. Okay. Uh-huh, yeah. That's safer. 'Cause then they learn the first time. Yeah, yeah. Okay. I really have to get out of here. So um what else is important before you run? Nothing, right? You got uh willing heads. Oh, I don't think I brought my diary up. Uh but maybe. Why? Okay. So Yeah. Mm-hmm. Hmm. So do you know what time of day you might want? 'Cause I should c I should check that. Okay. You all have to b you'll have to be there to when it falls over. Yeah. Okay. So that all sounds f good fun. Well I can just leave you guys here, you know. Why not.